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A COST-ANALYTIC APPROACH TO DETERMINING 
CHAPTER 1 PROGRAM IMPACT: SOME PRELIMINARY FINDINGS 

Abstract 



The paper reports an attempt to evaluate Instructional practices in 
compensatory education by looking at both costs and effects associated 
with the practices. The study covered some 300 Chapter 1 projects over a 
three-year period, focusing on project settings and instructional 
approaches used in the projects* Remits of the study indicated that the 
pull-out setting product the highest achievement gains and the most 
favorable cost -effectiveness ratio. A prescribed instructional system, 
supplemented by a miscellany of other materials, produced the highest 
achievement gains and the most favorable cost-effectiveness ratio. The 
cost information generated clearly provides a better basis for determining 
the relative worth of the various instructional settings and approaches 
used in compensatory education. 
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A Cost-Analytic Approach to Determining 
Chapter I ttrogiran Impact: Some Preliminary Findings 

INTRODUCTION 

Chapter 1 represents one of the largest federal investments in 
education for disadvantaged youth in the nation. Typically, Chapter 1 
projects provide remedial instructional services in basic skills areas 
such as reading and mathematics. In a majority of the projects, the 
services were provided in a pull-out setting (Stonehill and Anderson, 
1982) . Instructional approaches used in these projects are often 
described as "eclectic* involving the use of teacher-made as well as 
commercially packaged materials* The adoption or adaptation of 
instructional settings and approaches is often a matter of logistical 
convenience or fiscal necessity rather than an outcome of careful 
deliberations of benefits and impact. 

While there has been little direct evidence to indicate the efficacy, 
or the lack of it, of the pull-out setting, negative perceptions have 
been expressed by some researchers. Harnischfeger (1980), for example, 
pointed out two major shortcomings of the pull-out setting. First, 
pull-out teachers have little contact with the regular classroom 
teachers. As a result, there is little coordination or integration with 
regular classroom instruction and the regular classroom teachers lose 
close contact with the progress of students who are pulled -out. 
Secondly, when students receive reading instruction in a pull-out 
program, usually they are pulled-out of the regular reading instruction. 
Consequently, pull-out students do not receive extra learning opportunity 
and often receive less reading time than the rest of the class due to 
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increased transition time. On the other hand, since the pull-out setting 
often means smaller class size and generally provides greater 
instructional support to students, it could be expected to produce 
greater achievement (Tobias, 1982). Smith and Glass (1980) found that 
reducing class size had beneficial effects on both cognitive and 
affective outcomes. 

Compensatory education programs that appear different may not have 
differential impact upon participants. In a review of the effects of 
different curricula in early childhood education, Weikart (1981) found 
that the expected major differences between curricula simply were not 
obtained. The author suggested that the basic issues in successful 
programing had to do with the quality of program implementation rather 
than the philosophy of curriculum selected. 

In much the same vein, Tobias (1982) argued that external differences 
between instructional treatments, whether they are educational media, 
methods of organizing the classroom, or technological devices, are 
important only in terms of the degree to which they x- Luence the 
student's cognitive activities while engaged by the instructional 
content. External differences in instructional arrangements that lead to 
similar cognitive processing will result in comparable achievement, 
irrespective of superficial differences between methods. The author 
suggested that, generally, any instructional method or procedure which 
induces students to spend more time on task than a comparison mode will 
result in higher achievement. In the Instructional Dimensions Study 
(IDS) , Cooley and Leinhardt (1980) found that pull-out instruction was 
related inversely with reading achievement. However, in a reanalysis of 
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the same data involving only pull-out students, a positive relationship 
was evident between pull-out and posttest achievement in reading 
(Leinhardt and Pallay, 1982) . Teachers involved in the IDS project 
expressed mixed opinions about the pull-out setting. On the negative 
side, pull-out was said to be a managerial headache, demoralizing and 
disruptive to the children. On the positive side, it was seen as a way 
of providing intensive remediation to children who needed it. In an 
evaluation of the Emergency school Aid Act, Coulson et (1977) found 
that the pull-out setting was inversely associated with achievement. The 
authors reported that a change from receiving no pull-out instruction in 
reading to receiving one half reading instruction in a pull-out setting 
was associated with a reduction in reading achievement. 

Leinhardt and Pallay (1982) reported empirical work that demonstrated 
success, failure and no difference in a variety of instructional 
settings. The authors concluded that while setting was not an 
overwhelming variable in and of itself, it was worth investigating as a 
variable which influences features of instruction which affect student 
learning. 

Findings of no significant differences between instructional settings 
are consistent with Dahllof f s (1971) assertion that achievement cannot be 
regarded as a direct outcome of the grouping arrangement. As Leinhardt 
and Pallay (1982) pointed out, a setting does not eliminate or guarantee 
the presence of effective instructional practices. Most of the important c 
variables can occur in most settings. 

i 
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Program impact is, of course, only one side of the coin, the other 
side being program costs, if program impact has, de facto, become a 
constant (there being no significant differences), it would make sens* 
for program managers to look at cost as an important variable in deciding 
the relative merit of a program or a particular program component • The 
present study is an attempt to look at relative cost-effectiveness as an 
essential element of program evaluation* 

PROCEDURE 

The study grew out o* an increased awareness of information needs in 
making curriculiso decisions at Chapter 1 project schools in Hawaii. It 
was conducted jointly by an external researcher and internal project 
staff* The study consisted of a secondary analysis (Burstein, 197ft) of 
evaluation data gathered over a period of three years to address the 
* following research questions: 

1* What is the relative cost-effectiveness of different 

instructional settings in Chapter 1 projects? 
2. What is the relative cost-effectiveness of different 

instructional approaches in Chapter 1 projects? 
No experimental manipulation was used in the study. Data were coded 
from evaluation reports for some 300 Chapter 1 projects over a three-year 
period. Project settings and instructional approaches examined in the 
study were as follows: 
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Pull-out . This involved the pulling out of participating 
students from the r^gjlar classroom to receive remedial instructional 
services from the Chapter 1 teacher or teacher aide in a small group 
setting. 

Regular classroom . The setting required the Chapter 1 teacher 
or aide to intervene directly in the regular classroom, teacher 
or aide worked with project students while instruction was being 
provided to non-project students by the regular classroom teacher at 
the same time. 

Combination . This involved the provision of instructional 
services in a combination of pull-out and regular classroom settings. 

Instructional system . This consisted of the use of a prescribed 
system with a set of commercially packaged materials along with 
protocols for their use. 

Materials only . This approach allowed the Chapter 1 teacher or 
aide to use materials, both commercially packaged and teacher -made f 
in ways she/he saw fit without having to follow prescribed protocols. 

Combination. This consisted of the adaptation of a prescribed 
instructional system along with associated materials and the use of a 
miscellany of other materials, commercial or teacher -made , to 
supplement the prescribed system. 

Data sources were limited to evaluation reports prepared by external 
evaluators and related documents (e.g. f state directories and welfare 
reports) for the 1978-81 school years. 



5 8 



Data Coding 

Data coding was conducted by graduate students hired by the Hawaii 
state department of education. The coding covered some 300 plus 
evaluation reports and included such information as student grouping, 
diagnostic testing, inservice training, in addition to achievement gains, 
project settings, instructional approaches and project costs. With 
respect to the coding of instructional approaches a list of basal 
materials used in Chapter 1 projects was prepared by the compensatory 
education staff. The list was reviewed by language arts specialists at 
the state department of education. Based on this review, the three 
primary categories of instructional approaches mentioned earlier were 
established. 

As would be expected, a number of problems were encountered in coding 
information from the evaluation reports, including: 

Missing data . Information was not provided in the district or 
school-level reports. For example, no information on project 
settings and Instructional approaches was obtained for secondary 
projects for the 1978-79 and 1979-80 school years. 

Ambiguous narratives . Some project narratives in the 
school-level reports were difficult to interpret. For example, in 
some cases the project setting was not clearly described. 

Lack of discreteness Many projects used composite project 
settings and student groupings. For instance, some narratives 
mentioned use of individual, small group and large group instruction 
with various grouping configurations* 

Most problems were resolved on the basis of the data coders 9 best 
judgment, in other cases, the data sheets were left blank. 



Completed data sheets were key-punched and quality control measures 
were taken to ensure accuracy of coding and validity of the coded data* 
A few cases were discarded because of excessive missing data and some 
inaccuracies in coding were corrected. Data analyses were then performed 
on the "clean" data. Project schools were used as the unit of analysis. 
Initially, analyses were conducted by subject area (i.e. r reading and 
mathematics) • Due to the relatively small number of projects in 
mathematics , subsequent analyses were confined to reading projects. 

Student achievement was measured by means of the norm-referenced 
model of the Title I Evaluation and Reporting System (Tallmadge, et al.r 
1981) • The model provides a measure of achievement growth attributable 
to Chapter 1 projects on an equal-interval scale consisting of normal 
curve equivalents (NCEs) • Weighted means were calculated in aggregating 
data across grades (2-12) and across projects. 

The study took advantage of various streamlining methods in 
conducting a marginal cost anaysis (Levin, 1983) , Chapter 1 being a 
supplementary instructional program. Specifically, total "ingredients" 
cost was computed by determining the level of funding allocated by the 
state education agency to each project. Contributed inputs (e.g. , parent 
involvement) , carry-over funds and monies saved were assumed to be either 
negligible or comparable across projects. Per pupil cost was obtained by 
dividing the total "ingredients'* cot»~ for each project by the number of 
students participating in the project. A cost-effectiveness ratio was • 
computed by dividing per pupil cost by average NCR gains made by students 
receiving instruction under the various project settings and 
instructional approaches. 
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RESULTS 

Project Profile 

Based on descriptive statistics provided in the an^yies, project 
profiles were developed for Chapter 1 projects included in the study* 
The data indicate that for the three school years included in the study 
the average Chapter 1 reading project enrolled 98-127 students in five 
different grades and provided 96-99 hours of instruction annually. The 
project employed 3,4-4.9 PTE staff and produced an achievement gain of 
7-8 NCHs at a per pupil cost of $672-$712. 

A comparison suggests that while project features (e.g. , project 
enrolment, duration, hours of instruction and costs) fluctuated somewhat 
over the years, project impact in terns of NCE gains was quite stable, 
ranging from a low of 7.4 for the 1979-80 school year to a high of 8.2 
for the 1980-81 school year. Pretest status increased from 18.7 NCEs (7 
percentile) for the 1978-79 school year to 22.6 NCEr, (10 percentile) for 
the 1980-81 school year. On the other hand, increasingly less money was 
spent on the average student. The per pupil cost declined from $712 in 
1978-79 to $672 in 1980-81. 
Project Setting 

A majority (72 percent) of the projects provided instructional 
services in a pull-out setting. A small number (18 percent) provided 
instruction in the regular classroom. Some (10 percent) projects used a 
combination of settings (see Tabl't 1) . Results of the analysis suggest 
that the pull-out setting was a viable approach to providing Chapter 1 
services. Students who received instruction in this setting performed as 
well as, if not better than, their counterparts in other settings. 
Specifically, these students as a group made an NCE gain of 8.8 as 
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compared with gains of 6*3 and 7.5 in the regular classroom and the 
"combination" settings, respectively. 

The cost analysis revealed that the pull-out setting produced a unit 
of MCE gain for every $81 expended on project students. Corresponding 
figures for the regular classroom and the "combination" settings were $84 
and $86, respectively. Thus, the pull-out setting produced the highest 
MCE gains and had the most favorable cost-effectiveness ratio. 
Achievement gains made in the regular classroom setting were, on the 
average, lower than those made in the "combination" setting* However, it 
also cost a little less to produce a unit of NCE gain in the regular 
classroom setting than it did in the combination setting. 



Table 1 about here 



Instructional Approach 

Most (80 percent) of the projects used a combination of instructional 
approaches, i.e., materials with a prescribed instructional system as 
well as materials Without such a system. Some (14 percent) of the 
projects used only materials with a prescribed instructional system and a 
few (6 percent) used only materials without a prescribed system. The 
results in Table 2 suggest that a combination of approaches may well be 
superior to either approach. Students who received instruction in a 
combined approach made an average MCE gain of 9.4, comparing favorably 
with gain 1 and 7.1 under a prescribed instructional system and 

under "materials only," respectively. 



Table 2 about here 
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Results of fc&e cost-analysis also favor the "combination" approach, 
itie data show that this aporoach produced a unit of MCE gain for every 
$71 expended on project students. Corresponding figures for 
instructional systems and •instructional materials only" were $77 and 
$91, respectively. 

Thus, students who received Chapter 1 instruction under a combination 
of instructional approaches (e.g., a prescribed system supplemented by 
miscellaneous materials) were likely to make greater achievement gains 
than their counterparts under single approaches. The combination 
approach also produced the most favorable cost-effectiveness ratio. The 
"instruction .».. materials only " approach yielded the lowest average 
achievement tj<tin and the least favorable cost-effectiveness ratio. 



DISCUSSION 

Contrary to negative perceptions expressed by sosie researchers 
(Harnischfeger, 1980) , the pull-out setting was shown to have superior 
cost-effectiveness &&ong project settings exaained in the study. With 
regard to instructional approaches, the combination approach was shown to 
have worked as well as, if not better than, other approaches. In terns 
of achievement gain, students who received Chapter 1 instruction in the 
pull-out setting performed as well as, if not better than, their 
counterparts in other settings. This suggests that despite its apparent 
drawbacks (e.g., loss of regular classroom instruction, increased 
transition time) the pull-out setting remains a viable option for 
providing services to Chapter 1 youngsters. Students who received 
Chapter 1 instruction under a combination of instructional approaches 
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(e.g„, a prescribed system supplemented by miscellaneous materials) made 
as much, if not greater, achievement gains as their counterparts under 
single approaches. This perhaps points to the validity of allowing the 
teaching staff flexibility in using materials in ways most suitable for 
individual students in a project. 

Previous findings on the relationship between student achievement and 
expenditures receive support frca the present study. Pol ley (1976), for 
example, found a positive relationship between median teacher salaries 
and student achievement. Moreover, per pupil cost cm principals 1 
salaries was also positively related to achievement. Since the major 
share (over 80 percent) of Chapter 1 expenditures consists of personnel 
costs, the per pupil cost index used in the present study was highly 
similar to expenditure indices used in Polley's study. Cost indices used 
in the present study correlated positively (r * .37, p< .05) with 
achievement gains* Undoubtedly, this occurs because per pupil cost is a 
surrogate for program elements (e.g., more experienced teaching staff, 
better facilities and materials) which tend to produce high achievement. 

This finding is particularly pertinent in view of the current budget 
reductions in education programs. She cost cutting trends are prestaaably 
based on the belief that there is no relationship between expenditure and 
student achievement. This belief runs counter to the findings of the 
present study which indicate that higher per pupil costs not only result 
in greater achievement but do uo with more favorable cost-effectiveness 
ratios. 

During the course of the study, both the state and district 
compensatory education staff have indicated the importance of teacher 
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variables as contributing factors to project impact . It was pointed out 
chat the regular classroom setting had worked well in some schools where 
Chapter 1 teachers were well trained and where communication between the 
regular classroom teacher and Chapter 1 staff was effective. 
Furthermore, staff turnover was perceived to be a major factor affecting 
project impact* These teacher variables are undoubtedly important 
elements of a project and should receive attention in future studies of 
project impact. 

CONCLUDING REMARKS 

For too long educational researchers and evaluators have overlooked 
the importance of cost as a variable to be included in decisionmaking and 
policy formulation. Recent trends, however, suggest that both state and 
local education agency personnel as well as university researchers will 
be called upon to provide cost analysis information as a critical element 
of program evaluation. Cost analysis appears even more essential where 
research on teaching has shown non-significant differences between 
instructional treatments with respect to student outcomes* The present 
study provides an example of evaluation which looks at instructional 
processes in terms of cost as well as program impact. The cost 
information generated clearly provides a better basis for determining the 
relative worth of the various instructional settings and approaches. 
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Table 1 

Relative Cost-Effectiveness of Project Settings of Chapter 1 





Read! no Pro*i^rf«i 






Variable 


Regular 
Class room 


Pull-out 


Combination 


No. of Projects 


51 


207 


29 


Per Pupil Cost 


532 


713 


646 


NCE Gains 


6.3 


8.8 


7.5 


Cost Effectiveness 
Ratio 


84 


81 


86 




Table 2 






Relative Cost Effectiveness of Instructional Approaches 
of Chapter 1 Reading Projects (N-169) 


Variable 


Instructional 
Systems 


Instructional 
Materials Only 


Combination 


No. of projects 


23 


10 


136 


Per Pupil Cost 


622 


644 


699 


NCE Gain 


8.1 


7.1 


9.4 


Cost-Effectiveness 
Ratio 


77 


91 


71 
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